System and method for virtual reality data integration and visualization for 3d imaging and instrument position data

ABSTRACT

Systems and methods for virtual reality or augmented reality (VR/AR) visualization of 3D medical images using a VR/AR visualization system are disclosed that includes a computing device operatively coupled to a VR/AR device, which includes a holographic display and at least one sensor. The holographic display is configured to display a holographic image to an operator. The computing device is configured to receive at least one stored 3D image of a subject&#39;s anatomy and at least one real-time 3D position of at least one surgical instrument, to register the at least one real-time 3D position of the at least one surgical instrument to correspond to the at least one 3D image of the subject&#39;s anatomy, and to generate the holographic image comprising the at least one real-time position of the at least one surgical instrument overlaid on the at least one 3D image of the subject&#39;s anatomy.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of U.S. patent application Ser. No.15/918,418, filed Mar. 12, 2018, and issued as U.S. Pat. No. 10,258,426,the contents of which are hereby incorporated by reference herein intheir entirety. U.S. patent application Ser. No. 15/918,418 is acontinuation of PCT Application No. PCT/US2017/023221, filed Mar. 20,2017, which claims the benefit of U.S. Provisional Application No.62/310,969, filed Mar. 21, 2016, both of which are hereby incorporatedby reference herein in their entirety.

BACKGROUND

A central issue for interventional surgical procedures remainsvisualization of the unexposed anatomy and localization of medicaldevices within the organs and vessels, such as catheters, stents,probes, and the like. As procedures move away from maximal exposuretowards being minimally invasive, the requirements for enhancedvisualization are more profound. An example is minimally invasivetranscatheter ablation for cardiac arrhythmias.

In healthy hearts, organized electrical excitation causes heartcontraction. When this electrical activity becomes irregular, the heartcan no longer pump efficiently and patients experience dizziness,fainting, and/or sudden death. Statistics from the Centers for DiseaseControl and Prevention have estimated that in the United States suddencardiac death claims more than 600,000 victims every year. Erratic,irregular electrical cardiac activity is called an arrhythmia, and isoften caused by abnormal electrical connections in the heart. Cardiacarrhythmias affect people of all ages.

These short circuits can be effectively removed by applying one or moreenergy pulses to a selected region of the heart through a catheter thatis placed in the heart, known as a transcatheter ablation. Non-limitingexamples of types of energy pulses that may be applied using atranscatheter ablation include radiofrequency energy pulses, cryoenergypulses, and high frequency ultrasound pulses. A mainstay of modernarrhythmia therapy, ablation procedures require multiple catheters to beinserted into the heart to record electrical activity, identify keylocations responsible for the arrhythmia, and ablate tissue using eitherradiofrequency energy or cryotherapy. Currently, ablation procedures arecomplicated by the masking of the heart by the chest wall and separationof data (i.e., electrical signals, anatomic location, etc.) in theelectrophysiology laboratory, requiring the physician to mentallyreconstruct a heart model.

These procedures have been enhanced significantly by the development ofelectroanatomic mapping systems that construct a point-by-point map ofthe interior surface of the heart (endocardium) incorporating bothanatomic location and the local electrical signal. However, thesesystems are limited by the display of key measurements on multipletwo-dimensional screens. The skill to mentally relate electricalrecordings to the overall multi-dimensional cardiac anatomy remains akey challenge in the training of cardiac electrophysiologists andintra-procedural collaboration. It is therefore intensely difficult totrain new physicians, and significant skill-dependent variability inoutcomes is common.

SUMMARY

In one aspect, a VR/AR visualization system is provided. The VR/ARvisualization system includes a VR/AR device that includes a holographicdisplay configured to display a holographic image to an operator, and acomputing device operatively coupled to the VR/AR device. The computingdevice includes a non-volatile memory and a processor. The computingdevice is configured to receive at least one stored 3D image of asubject's anatomy, to receive at least one real-time 3D position of atleast one surgical instrument, to register the at least one real-time 3Dposition of the at least one surgical instrument to correspond to the atleast one stored 3D image of the subject's anatomy; and to generate theholographic image. The holographic image includes the at least onereal-time 3D position of the at least one surgical instrument overlaidon the at least one 3D image of the subject's anatomy.

In another aspect, a method of VR/AR visualization of 3D medical imagesis provided. The method includes receiving, using a computing device, atleast one stored 3D image of a subject's anatomy. The computing deviceis operatively coupled to a VR/AR device, and the VR/AR device includesa holographic display and at least one sensor. The method furtherincludes receiving, using the computing device, at least one real-time3D position of at least one surgical instrument, registering the atleast one real-time 3D position of the at least one surgical instrumentto correspond to the at least one stored 3D image of the subject'sanatomy; and displaying, using the holographic display, a holographicimage comprising the at least one real-time 3D position of the at leastone surgical instrument overlaid on the at least one 3D image of thesubject's anatomy to an operator.

In an additional aspect, at least one non-transitory computer-readablestorage media for providing VR/AR visualization of three-dimensionalmedical images to an operator is provided. The computer-readable storagemedia has computer-executable instructions embodied thereon, wherein,when executed by at least one processor, the computer-executableinstructions cause the processor to receive at least one stored 3D imageof a subject's anatomy, receive at least one real-time 3D position of atleast one surgical instrument, register the at least one real-time 3Dposition of the at least one surgical instrument to correspond to the atleast one stored 3D image of the subject's anatomy, and display aholographic image. The holographic image includes the at least onereal-time 3D position of the at least one surgical instrument overlaidon the at least one 3D image of the subject's anatomy to an operator.

In an aspect, the disclosure is a virtual reality/augmented reality(VR/AR) system for procedures that occur within portions of a patient'sbody. In one aspect, the disclosure is a VR/AR system for medicalprocedures that occur within hard to view, as well as access, portionsof the human anatomy. In an aspect, the VR/AR system provides amulti-dimensional experience for a user (e.g., an operator) during theprocedures, with a patient-specific 3D representation of the patient'sinternal human organ/system in addition to other procedure relevantdata. In an aspect, the 3D representation and additional information canbe presented in an augmented reality environment. In other aspects, suchinformation can be provided in a VR/AR environment. In addition, thesystem is able to map and represent the real-time positioning of aninstrument (e.g. a catheter) used during the procedure. The 3Drepresentation and the relevant data are configured to be presented andallow interaction with the user such that the user does not need tocommunicate with any other person nor break sterility. For instance, theuser can provide commands to the system without physically contacting aninput device with any part of the user's body such as the user's hands.

In an exemplary aspect, the VR/AR system is for cardiac interventionalprocedures. The system provides a patient-specific 3D model of the heartof the patient in real time as the procedure is occurring, including theability to track positioning of catheters used within the patient'sheart. Additional information can be provided to the user through othersenses (e.g., auditory signals) in a manner as described below.

In one aspect, the VR/AR system receives, using a computing device, atleast one stored 3D image of a subject's anatomy (e.g. a patient'sanatomy). The computing device is coupled to a VR/AR device thatcomprises a holographic display and at least one sensor. The computingdevice receives at least one real-time 3D position of at least onesurgical instrument. The at least one real-time 3D position of at leastone surgical instrument is registered to correspond to the at least onestored 3D image of a subject's anatomy. The holographic display displaysa holographic image comprising the at least one real-time 3D position ofat least one surgical instrument overlaid on the at least one 3D imageof a subject's anatomy to an operator.

In another aspect, the VR/AR system generates a first 3D image of anelectroanatomic visualization representing a cardiovascular organ of asubject in 3D by processing a first set of sensor data generated by acatheter inserted inside the cardiovascular organ. The first 3D image isprovided to a holographic display including, but not limited to, ahead-mounted display (HMD) worn by an operator to display theelectroanatomic visualization in a field of view of the operator. Asecond set of sensor data is received from an input device, where thesecond set of sensor data is indicative of motion of a body part of theoperator to interact with the electroanatomic visualization. A path ofthe motion of the body part is determined by processing the second setof sensor data. An angle of rotation for the electroanatomicvisualization is determined based on the path of motion. A second 3Dimage is provided to the HMD to update the display of theelectroanatomic visualization by rotating the cardiovascular organ bythe angle of rotation in the field of view of the operator.

These and other objects and advantages of the invention will becomeapparent from the following detailed description of the preferredembodiment of the invention. Both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are intended to provide further explanation of the invention asclaimed.

The accompanying drawings are included to provide a furtherunderstanding of the disclosure and are incorporated in and constitutepart of this specification, illustrate several embodiments of thedisclosure and together with the description serve to explain theprinciples of the disclosure.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a fluoroscopic image used to localize catheters within theheart according to an existing method.

FIG. 2 is a schematic representation of a VR/AR system for internalmedical procedures according to one aspect of the disclosure.

FIG. 3 is an image of an electroanatomic map according to an aspect ofthe present disclosure.

FIG. 4 illustrates a 3D model of a patient's heart according to anaspect of the present disclosure.

FIG. 5A is an image of a view provided to the operator of the VR/ARsystem that includes a drop-down menu according to one aspect.

FIG. 5B is an image of a view provided to the operator of the VR/ARsystem that includes an inset image drop-down menu according to oneaspect.

FIG. 5C is an image of a view provided to the operator of the VR/ARsystem that includes a cross-sectional view of a 3D heart modelaccording to one aspect.

FIG. 6 is a block diagram showing a schematic representation ofcomponents of the VR/AR system of FIG. 2.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the preferred embodiments,reference is made to the accompanying drawings, which form a parthereof: and within which are shown by way of illustration specificembodiments by which the disclosure may be practiced. It is to beunderstood that other embodiments may be utilized and structural changesmay be made without departing from the scope of the disclosure.

As used in the specification and the appended claims, the singular forms“a,” “an” and “the” include plural referents unless the context clearlydictates otherwise. Ranges may be expressed herein as from “about” oneparticular value, and/or to “about” another particular value. When sucha range is expressed, another embodiment includes from the oneparticular value and/or to the other particular value. Similarly, whenvalues are expressed as approximations, by use of the antecedent“about,” it will be understood that the particular value forms anotherembodiment. It will be further understood that the endpoints of each ofthe ranges are significant both in relation to the other endpoint, andindependently of the other endpoint.

“Optional” or “optionally” means that the subsequently described eventor circumstance may or may not occur, and that the description includesinstances where said event or circumstance occurs and instances where itdoes not.

Throughout the description and claims of this specification, the word“comprise” and variations of the word, such as “comprising” and“comprises,” means “including but not limited to,” and is not intendedto exclude, for example, other additives, components, integers or steps.“Exemplary” means “an example of” and is not intended to convey anindication of a preferred or ideal embodiment. “Such as” is not used ina restrictive sense, but for explanatory purposes.

Disclosed are components that can be used to perform the disclosedmethods and systems. These and other components are disclosed herein,and it is understood that when combinations, subsets, interactions,groups, etc. of these components are disclosed that while specificreference of each various individual and collective combinations andpermutation of these may not be explicitly disclosed, each isspecifically contemplated and described herein, for all methods andsystems. This applies to all aspects of this application including, butnot limited to, steps in disclosed methods. Thus, if there are a varietyof additional steps that can be performed it is understood that each ofthese additional steps can be performed with any specific embodiment orcombination of embodiments of the disclosed methods.

As will be appreciated by one skilled in the art, aspects of the currentdisclosure may take the form of an entirely hardware embodiment, anentirely software embodiment, or an embodiment combining software andhardware aspects. In an aspect, the current disclosure can include acombination of physical components configured to perform certain stepsand functions (e.g., obtaining electroanatomical measurements, etc.)that are controlled by a combination of hardware and softwarecomponents. Furthermore, components of the methods and systems may takethe form of a computer program product on a computer-readablenon-transitory storage medium having computer-readable programinstructions (e.g., computer software) embodied in the storage medium.Any suitable computer-readable storage medium may be utilized includinghard disks, CD-ROMs, optical storage devices, flash storage devices,solid state storage devices, and magnetic storage devices.

Further, components and methods utilized by the disclosure as describedbelow can be performed in a program environment, which may incorporate ageneral purpose computer or a special purpose device, such as a hardwareappliance, controller, or handheld computer. In addition, the techniquesof the components described herein can be implemented using a variety oftechnologies known in the art. For example, the methods may beimplemented in software executing on a computer system, or implementedin hardware utilizing either a combination of microprocessors or otherspecially designed application specific integrated circuits,programmable logic devices, or various combinations thereof.

Some aspects of the methods and systems are described below withreference to block diagrams and flowchart illustrations of methods,systems, apparatuses and computer program products. It will beunderstood that each block of the block diagrams and flowchartillustrations, and combinations of blocks in the block diagrams andflowchart illustrations, respectively, can be implemented by computerprogram instructions. These computer program instructions may be loadedonto a general purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions which execute on the computer or other programmabledata processing apparatus create a means for implementing the functionsspecified in the flowchart block or blocks.

These computer program instructions may also be stored in anon-transitory computer readable memory that can direct a computer orother programmable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including computer readableinstructions for implementing the function specified in the flowchartblock or blocks. The computer program instructions may also be loadedonto a computer or other programmable data processing apparatus to causea series of operational steps to be performed on the computer or otherprogrammable apparatus to produce a computer-implemented process suchthat the instructions that execute on the computer or other programmableapparatus provide steps for implementing the functions specified in theflowchart block or blocks.

Accordingly, blocks of the block diagrams and flowchart illustrationssupport combinations of means for performing the specified functions,combinations of steps for performing the specified functions, andprogram instruction means for performing the specified functions. Itwill also be understood that each block of the block diagrams andflowchart illustrations, and combinations of blocks in the blockdiagrams and flowchart illustrations, can be implemented by specialpurpose hardware-based computer systems that perform the specifiedfunctions or steps, or combinations of special purpose hardware andcomputer instructions.

In various aspects, the disclosure is directed to a virtual realityand/or augmented reality (VR/AR) system, discussed in detail below.While the term virtual reality is used throughout to describe thesystem, a person of skill in the art would understand that in thegeneric use of the term, virtual reality may include virtual reality(VR) and augmented reality (AR). In some instances, the term “VR/AR”will be used to identify the system. Therefore, when the term “virtualreality” or “VR/AR” is used herein, it should be understood to encompassall types of modified realities, unless specifically distinguished.

“Virtual reality”, as used herein, refers to a method of displayingand/or interacting with one or more elements representingcomputer-generated data. Typically, all elements visible within a fieldof view of a virtual reality display are computer-generated elements.

“Augmented reality”, as used herein, refers to a method of displayingand/or interacting with one or more elements representingcomputer-generated data. Augmented reality is a blend of virtual realityand real life, wherein a typical augmented reality display includes oneor more computer-generated elements overlaid over the real-life objectsvisible by an operator. The term “augmented reality”, as used herein,may further include “mixed reality”, a term that refers to the augmentedreality display method that specifically includes an ability for a useror operator to interact with the computer-generated elements.

“Holographic display”, as used herein, refers to a method of displayingand/or interacting with a virtual 3D object in which the virtual 3Dobject is dynamically updated to modify an operator's view of thevirtual 3D object in response to movements of the operator, or operatorrequested modifications of the view of the virtual 3D object, such asmagnified/reduced, translated/rotated, cross-sectioned views of thevirtual 3D object, etc.

The disclosure is directed to a VR/AR visualization system 10 (alsoreferred to herein as a VR/AR system 10 or system 10) for visualizationand manipulation of 3D imaging data as well as additional informationincluding, but not limited to, 2D imaging data, vital sign data, andsubject demographic data in association with medical diagnostic andtreatment procedures that occur within hard to view/access portions of asubject's anatomy. By way of non-limiting example, the VR/ARvisualization system 10 can be utilized for procedures within the heart,gastrointestinal system, ear canal, and other types of anatomy orbiological systems. While the embodiments described below are directedto a VR/AR visualization system 10 associated with cardiacinterventional procedures, one skilled in the art would recognize thatusing other 3D localization or pose estimation modalities, including,but not limited to, impedance-based localization, magnetic localization,marker-based localization, or 3D ultrasound in combination with theVR/AR visualization system 10 described herein could be readily extendedto diagnostic or interventional procedures in other organ systems and/orother anatomical regions.

In one aspect, the disclosure is directed to a VR/AR visualizationsystem 10 for use in association with cardiac diagnostic proceduresand/or cardiac interventional procedures. The VR/AR system 10, via a 3Dmedical imaging device 40, is able to capture the anatomical features ofa heart 15 of a patient 20 (e.g. a subject). In addition, electricaldata of the heart 15 can be obtained using one or more electroanatomicmapping devices 50, which are configured to collect electrical data aswell as the positioning of associated instruments and the location ofthe electrical data measurement. Non-limiting examples of instrumentsassociated with the one or more electroanatomic mapping devices 50include diagnostic catheters, reference catheters, ablation catheters,monitor catheters, noncontact mapping catheters, multielectrode arraycatheters, multipolar catheters, multipolar circular mapping catheters,and magnetic sensor catheters. In one aspect, data obtained by the oneor more electroanatomic mapping devices 50 may be analyzed by anassociated electroanatomic mapping system to determine an endocardialmap describing the spatial arrangement of the interior surfaces of theatria and ventricles of a heart 15, as well as the spatial arrangementof veins and arteries in relatively close proximity to the heart 15including, but not limited to a vena cava, an aorta, a pulmonary artery,and the like.

In an aspect, one or more electroanatomic mapping devices 50 may providefor combining at least one 3D map of anatomical features of the heart 15of the patient 20, obtained using the 3D medical imaging device 40, andat least one 3D endocardial surface map obtained using the one or moreelectroanatomic mapping devices 50. In one aspect, the 3D coordinatesystem within which the 3D endocardial surface map is defined may beregistered to the coordinate system within which the 3D map ofanatomical features of the heart are defined, so that the 3D map ofanatomical features and the 3D endocardial surface map are definedwithin the same 3D coordinate system.

In an aspect, data defining the co-registered 3D map of anatomicalfeatures and 3D endocardial surface map produced by the one or moreelectroanatomic mapping devices 50 may be received by a computing device30 of the VR/AR visualization system 10. Using the data defining theco-registered 3D map, the computing device 30 generates a 3D model 65 ofthe heart 15 configured to be displayed within a holographic imageproduced by the holographic display of a VR/AR device 60.

In one aspect, the holographic image may consist solely of at least aportion of the 3D model 65 of the heart 15. In various other aspects,the computing device 30 of the VR/AR visualization system 10 may receiveadditional data that may be incorporated into the holographic image fordisplay to the operator on the holographic display of the VR/AR device60.

In various aspects, the VR/AR system 10 may include multiple VR/ARdevices 60. By way of non-limiting example, a first operator may wear afirst VR/AR device 60 including, but not limited to, a firsthead-mounted display, and a second operator may wear a second VR/ARdevice 60 including, but not limited to, a second head-mounted display.In this non-limiting example, the first and second operators may performa surgical procedure together on a patient. The first and second VR/ARdevices 60 may display different views of the same organ or portion ofanatomy of the patient 20. For instance, displayed views of thepatient's heart 15 may positioned at different angles based on thelocation of the corresponding VR/AR device 60 relative to the patient20.

In various other aspects, additional data may be received by thecomputing device 30 of the VR/AR visualization system 10 andincorporated into the holographic image. Non-limiting examples ofadditional data suitable for incorporation into the holographic imageinclude: real-time 3D data defining positions and/or measured valueswithin the coordinate system defining the 3D model 65; real-time 2D dataproduced by 2D imaging devices such as fluoroscope imaging devices;real-time numerical measurements such as one or more vital signs,pre-determined data such as patient demographic data, and anycombination thereof. In these various other aspects, the additional datamay be incorporated and/or overlaid on the 3D model 65 of the heart 15,or the additional data may be displayed as a separate element within theholographic image as described in additional detail below.

In one aspect, additional data obtained by the electroanatomic mappingdevice 50 may be received by the computing device 30 and incorporatedinto the holographic image. In one aspect, the electroanatomic mappingdevice may further include an instrument position sensor configured toobtain at least one real-time 3D position of at least one surgicalinstrument, including, but not limited to, an electrophysiology (EP)catheter. In this aspect, computing device 30 may receive the at leastone real-time 3D position of the at least one surgical instrument andgenerate the holographic image that includes the at least one real-timeposition of the at least one surgical instrument overlaid on the atleast one 3D image of the subject's anatomy, including, but not limitedto, the 3D model 65 of the heart 15. In another aspect, the at least onereal-time 3D position of the at least one surgical instrument may beobtained using a separate device including, but not limited to aseparate instrument position sensor of a surgical instrument system.Non-limiting examples of suitable instrument position sensors includeone or more electroanatomic mapping devices in addition to theelectroanatomic mapping device 50 used to obtain the 3D endocardialsurface mapping data, and other real-time position mapping systems thatinclude position sensing devices that make use of ultrasound, magneticfields, electrical fields, and/or any other existing suitable positionsensing method.

In another aspect, additional data obtained by the electroanatomicmapping device 50 may include additional electrophysiologicalmeasurements obtained by one or more electrophysiology (EP) catheters.In one aspect, the data may include additional 3D real-time datadefining one or more datasets of real-time electrophysiologicalmeasurements mapped to the coordinate system of the 3D model 65 of theheart 15. Non-limiting examples of real-time electrophysiologicalmeasurements include voltage maps, activation timing maps, andpropagation maps. In this one aspect, the maps of additionalelectrophysiological measurements received by the computing device 30may be overlaid on the 3D model 65 of the heart 15 within theholographic image. In an aspect, the maps of additionalelectrophysiological measurements may be selected for display, to berendered transparent, or to be removed from the holographic image asdefined by one or more cues produced by the operator. In another aspect,additional data obtained by the electroanatomic mapping device 50 mayinclude additional electrophysiological measurements associated withablation by an ablation catheter including, but not limited to, anamount of radiofrequency (RF) or cryoenergy applied at a particularposition within the 3D model 65 of the heart 15. In this other aspect,the additional electrophysiological measurements may be incorporatedinto the holographic image in the form of a 3D visual element, such as acircle or other symbol positioned at the point of application of theablation energy within the 3D model 65 and/or a color, size, numericalvalue or other visual element to indicate the amount and/or direction ofthe ablation energy or ablation force measured during ablation eventsenabled by the ablation catheter.

In an additional aspect, the computing device 30 may receive one or moreadditional datasets defining at least one additional 2D image obtainedfrom at least one additional medical imaging device. In one aspect, theat least one additional 2D image may include a 2D representation of a 3Dreal-time image including, but not limited to, a real-time fluoroscopeimage. In this one aspect, the at least one additional 2D image may beincorporated into the holographic image in the form of a 2D visualelement displayed separately from the 3D model 65. Non-limiting suitable2D visual elements incorporated into the holographic image include avirtual 2D monitor, an inset image, and any other known representationof a 2D visual element in a 3D holographic image. By way of non-limitingexample, an additional dataset defining a real-time fluoroscopic imagemay be incorporated into the holographic image in the form of an insetimage, as illustrated in FIG. 5B.

In one aspect, the holographic image may be displayed on a holographicdisplay of a VR/AR device 60 to an operator. In response to one or morecues from the operator, the computing device 30 may modify theholographic image as displayed on the holographic display according tothe preferences of the operator. By way of non-limiting example, theoperator may enlarge, reduce, rotate, or move the holographic image asdisplayed on the holographic display to facilitate the accomplishment ofa diagnostic and/or surgical procedure. In various aspects, if the VR/ARsystem 10 includes multiple VR/AR devices 60, a first VR/AR device 60worn by a first operator may be operatively coupled to the computingdevice 30 such that only the first operator may modify the holographicimage as displayed on all holographic displays of all VR/AR devices 60of the VR/AR system 10. In these various other aspects, the computingdevice 30 may receive the relative positions and orientations of each ofthe multiple VR/AR devices 60 and generate a holographic image for eachVR/AR device 60 corresponding to each position and each orientation ofeach VR/AR device 60 relative to the first VR/AR device 60 worn by thefirst operator, who also controls modifications of the holographicimage.

In one non-limiting example, the computing device 30 may generate anelectrocardiogram of a patient's heart for display in an inset image ofthe holographic image. The computing device 30 may receive an indicationof a user input (i.e. a cue) from the VR/AR device 60 or a separateinput device (e.g., a camera or motion sensor) to position virtualmarkers on the electrocardiogram without breaking sterility. Based onthe user input, the computing device 30 may determine a metric fordisplay on the inset image along with the electrocardiogram. Forinstance, the virtual markers may be virtual calipers configured tomeasure a period or amplitude of the patient's heartbeat (e.g. ametric).

In another additional aspect, the computing device 30 may receive one ormore additional alphanumeric datasets including, but not limited to, apatient demographic dataset and/or real-time measurements of vital signsof the subject obtained from existing vital sign measurement devices. Inthis other aspect, the one or more additional alphanumeric datasets maybe incorporated into the holographic image in the form of a transparentalphanumeric element overlaid within the holographic image.

In additional aspects, the computing device 30 may generate a menu orother image element that includes user-selectable elements displayed inthe form of a non-transparent or transparent row or column ofalphanumeric strings, symbols, and/or icons. In these additionalaspects, the user-selectable elements provide a means of selecting oneor more instructions to be executed by one or more processors of thecomputing device 30 to enable the operation of the VR/AR system 10. Byway of non-limiting example, the computing device 30 may generate atransparent menu within the holographic image as illustrated in FIG. 5A.

By selecting one or more of the interactive menu elements within theholographic image, the operator of the VR/AR device 60 can also view andinteract with the 3D model 65 and additional information without havingto break sterility or communicate with anyone present during adiagnostic or surgical procedure. As disclosed herein, the VR/AR device60 can present and remove the information as needed in a true virtualreality environment or an augmented reality environment depending on theneeds of the operator of the system 10. In one aspect, an arrangement ofelements within the holographic image may be saved by the computingdevice 30 and retrieved by the VR/AR system 10 for subsequent use by thesame operator as a preferred arrangement of elements.

In various aspects, the computing device 30 may update the holographicimage over time to account for changes due to one or more time-varyingfactors including, but not limited to: section or deselection of auser-selectable menu element by the operator, receipt of updatedreal-time data such as updated vital signs data, an additional ablationevent, a change in other datasets, such as the real-time datasetassociated with the fluoroscope imaging. The computing device 30 mayupdate the holographic image to include a representation of a portion ofan instrument positioned relative to a patient's organ, e.g., a positionof a catheter tip relative to the patient's heart. The computing device30 may update the holographic image based on sensor data from theinstrument indicative of detected motion of the instrument.

In an aspect, the computing device 30 of the VR/AR system 10 isconfigured to receive a reconstructed 3D model 65 of the patient'scardiac anatomy. Here, the 3D medical imaging device 40 is configured toacquire specific physical information related to the patient's heart 15.The information will encode the dimensions of the heart 15 and itscomponents/cavities and pre-defined anatomical landmarks. Theseanatomically distinct landmarks are used to register the 3D model 65 tothe electroanatomic map of the endocardial surface obtained by theelectroanatomic mapping device 50. By registering the electroanatomicmap to the coordinate system within which the 3D anatomical model, othermeasurements obtained by the electroanatomic mapping device 50 that aremapped to the electroanatomic map of the endocardial surface are alsoregistered to the coordinate system defining the 3D model 65 of theheart 15, enabling these mapped electrophysiological measurements to bevisualized with respect to the 3D model 65 within the holographic imagedisplayed to the operator. In some aspects, the 3D model 65 and theelectroanatomic map are generated in different physical units. Thus, tonormalize to the same coordinate system, the computing device 30registers (e.g., maps) a first set of points of the 3D model 65 to asecond set of points of the electroanatomic map, e.g., based on theanatomical landmarks.

In various aspects, the registration of the electroanatomic map of theendocardial surface to the 3D anatomical model obtained using the atleast one 3D medical imaging device is performed by the computing device30 and/or the electroanatomic mapping device 50. The registration of thetwo maps may be performed using any suitable existing method withoutlimitation. In one aspect, at least several pre-determined landmarks areidentified in the 3D anatomical model and in the 3D endocardial surfacemap to provide defined points of intersection of the two maps.Non-limiting examples of suitable cardiac landmarks include the superiorvena cava/right atrial junction, the inferior vena cava/right atrialjunction, the coronary sinus, the right ventricular apex, and the rightventricular outflow tract. While landmarks associated with other knownsections of the heart can be used for landmarks in other aspects of thedisclosure, these landmarks are easy to access and identify, via Milland fluoroscopic processes, and registered with the mesh, as discussedin more detail below. In other aspects, statistical or feature-basedinference can be used to determine the location and number of points.

In an aspect, the 3D medical imaging device 40 can include any devicethat is capable of capturing 3D data related to the anatomy of theheart. For example, the 3D medical imaging device can include, but isnot limited to, fluoroscopic devices, echocardiogram devices (e.g.,transthoracic, transesophageal, intracardiac), X-ray devices,exploratory endoscopic systems, MRI and CT scanners, and the like. A 3Dmedical imaging device 40 may be selected depending on the type ofdiagnostic and/or surgical procedure to be performed in combination withthe VR/AR visualization system 10. For example, an echocardiogram device40 provides for a rapid acquisition of cardiac anatomy because it isnon-invasive, can be done quickly, does not expose the patient toradiation, and is anesthesia-free. Further, there is not a prolongedrequirement of immobility of the subject. In one aspect, data definingthe reconstructed 3D anatomical model may be obtained and analyzed bythe 3D medical imaging device 40 prior to a diagnostic or surgicalprocedure performed in combination with the use of the VR/ARvisualization system 10.

The 3D medical imaging device 40 captures the spatial information usedto reconstruct the 3D model 65 of the subject's heart 15. In one aspect,the 3D medical imaging device 40 reconstructs the spatial data andcreates a 3D anatomical model of the subject's cardiac anatomy. Varioustechniques known in the art can be used to create the 3D anatomicalmodel. By way of non-limiting example, multiple 2D views of thepatient's heart can be collected and then re-assembled into a coarse 3Dimage of the patient's anatomy. Once the 3D image of the patient'sanatomy is generated, the electroanatomic mapping device 50 provides, tothe computing device 30, a generic high resolution heart mesh to bemodified and/or transformed to match the overlap of the respectivelandmarks identified in the anatomical heart model and theelectroanatomical endocardial surface model. The computing device 30transforms the heart mesh using the data captured from the created 3Dimage of the patient's cardiac anatomy, including the spatial dimensionsof the captured landmarks (discussed above). Once transformed, a full 3Dmodel 65 of the cardiac structure of the patient is generated by thecomputing device 30 or the electroanatomic mapping device 50. By way ofone non-limiting example, the 3D medical imaging device 40 may create a3D anatomical model by generating a series of connected 2D polygons(e.g., a mesh) to represent 3D geometries of the corresponding anatomy.By way of another non-limiting example, the device 40 may create a 3Danatomical model using ray casting and/or tracing.

After the transformed full 3D model 65 of the patient's cardiacstructure has been created at the electroanatomic mapping device 50,electroanatomic data, including electrophysiological data specific forthe ablation needs are collected from the subject's and mapped to thetransformed full 3D model 65. In an aspect, the electroanatomic data iscollected from one or more electroanatomic mapping devices 50. In anaspect, the electroanatomic mapping devices 50 include electrophysiology(EP) catheters that are placed within the patient's heart 15. Themeasured electroanatomic data can include the electric activity (e.g.,voltage data, activation timing, and propagation maps) that is occurringat a given location of the patient's heart 15, which can be used todetermine where ablation needs to occur using existing diagnosticmethods.

To perform an EP study, the VR/AR system 100 may use one or moreelectroanatomic mapping devices 50. In one aspect, diagnostic catheters50 may be used initially, and an additional ablation catheter 50 may beused if ablation is indicated based on the diagnosticelectrophysiological measurements. In addition, any energy applied by anablation catheter 50 may be measured and recorded by the one or moreelectroanatomic mapping devices 50. By way of non-limiting example,ablation catheters may be configured to apply radiofrequency orcryoenergy at the catheter's tip. In another aspect, an ablationcatheter may be configured to sense the level of force applied by thecatheter (including directionality i.e. axial or lateral) at the tip ofthe catheter 50. Such information can be used to model the actual lesioncreation in the virtual heart of the 3D model 65 as the lesion is beingmade in real-time by using measured force and impedance. In one aspect,the force data measured from the ablation catheter can then be used forauditory feedback to the operator with the VR/AR device 60. By way ofnon-limiting example, the VR/AR device 60 may provide auditory feedbackincluding varying pitch and frequency of tones, indicative of the forcelevel of the ablation catheter 50 pushing on the cardiac tissue.

The EP catheters 50 may be moved by the operator throughout thepatient's heart 15 and collect diagnostic information. Both diagnosticand ablation catheters 50 are configured to obtain electroanatomic datathroughout their use, including during ablation. In addition tocollecting the electroanatomic data, the VR/AR system 100 may alsocapture the spatial related information, i.e., where the electroanatomicdata is occurring within the patient's heart, so that theelectroanatomic data can be mapped to the transformed full 3D model. Inan aspect, the electroanatomic mapping device 50 captures thepositioning of instruments inside a patient as well. For example, whenEP catheters 50 are employed, the positioning of the electrodes and thedistal part of the shaft of the catheter 50 are captured. In an aspect,the electroanatomic mapping device 50 utilizes an electroanatomicmapping system to find the coordinates of the electroanatomic data. Themapping system is able to identify the X, Y, and Z coordinates of the EPcatheters 50 within the heart 15, and then can place the electroanatomicdata to the coordinates. For example, an electroanatomic mapping systemsuch as ENSITE™ NAVX™ Navigation and Visualization and CARTO™ (BiosenseWebster Systems) can be used to collect these data. However, othersystems capable of providing such information can be used.

FIG. 3 illustrates an example of a current electroanatomic map for usein a cardiac model. This cardiac model is limited to the geometries ofthe right atrium (RA) and left atrium (LA). The image on the left ispresented in orthogonal views, wherein a right anterior oblique (RAO)view is on the left and a long axial oblique (LAO) view is on the right.The distal ends of the electrophysiologic catheters are visualizedinside this geometry. The first (1st) catheter, having four electrodes,is positioned at the site of normal conduction, at the His location. The2nd catheter, having 10 electrodes, is positioned in the coronary sinus.The 3rd catheter, having 4 electrodes, is only visualized in the LAOprojection as it is advanced through the tricuspid valve and into theright ventricle. Lastly, the radiofrequency (RF) ablation catheter,having 4 electrodes, is positioned at the site of abnormal electricaltissue. The spheres (shown in red) mark the sites where RF lesions wereplaced. The bullseye projection in the center bottom portion of FIG. 3indicates the total force (TF) applied by the ablation catheter tocardiac tissue.

While the above describes electroanatomic mapping devices 50 andelectroanatomic data associated with cardiac procedures, it isunderstood that the devices and data can be associated with othersystems and organs found within the human body.

As the electroanatomic data is collected and mapped as described above,the VR/AR device 60 is provided with a 3D model 65 (see FIG. 4) of thepatient's heart 15 by the computing device 30, including the placementof the EP catheters 50 (see FIG. 5B). In an aspect, the VR/AR device 60can include a true virtual reality (VR) device (i.e., a device thatfully immerses the operator in the created environment) or an augmentreality (AR) device (i.e., operator can have images or models displayedvirtually in the virtual space, but is still able to see and interactwith the true environment). Along those lines, the VR/AR device 60 canbe a wearable device, including, but not limited to, the MicrosoftHOLOLENS' (i.e. an AR device) and OCULUS RIFT™ (i.e. VR device). TheVR/AR device 60 may include sensors such as motion sensors (e.g.,accelerometers, gyroscopes, or inertial measurement units), audiosensors, eye and gaze tracking sensors, and/or an electronic display,among other components. In another aspect, the VR/AR device 60 canprovide a projected holographic display that includes the 3D model 65.The VR/AR device 60 may be communicatively coupled to the HMD via awireless exchange protocol, or via a wired connection. In at least someaspects, use of an AR device may be advantageous since it allows theoperator to see the patient and interact with the patient in real timewhile simultaneously viewing and interacting with the 3D model 65 andderiving the benefits of it, making for a safer patient experience.

In such aspects, the 3D model 65 within the holographic image may berendered as obtuse or semi-transparent depending on the location of theEP catheters, or fully transparent to enable an unimpeded view of the EPcatheter positions. In a transparency view, the operator may change thetransparency of the cardiac walls using operator-enabled cues receivedby at least one sensor of the VR/AR device, allowing for readilyapparent visualization of the catheters in the heart during a diagnosticand/or surgical procedure. The portions of the catheter can also berepresented throughout any view. In addition, the VR/AR device 60 canalso allow the operator to manipulate the position, orientation, andsize of the heart, as well as to create slices to view. Also, theoperator can switch between views, as well as data display, without theuse of hands so the operator can maintain sterility throughout theentirety of the procedure.

By way of non-limiting example, the VR/AR system 100 may use head and/oreye tracking technology to receive input commands (e.g., user input)from the operator without requiring the operator to physically touch aninput device (e.g., VR/AR device 60) using the operator's hands. In someembodiments, the input device is physically connected to the VR/ARdevice 60. In other embodiments, the input device is separate from theVR/AR device 60 and communicatively coupled to the computing device 30.For example, the input device is a Microsoft KINECT′. The input devicemay include imaging sensors (e.g., cameras), illumination sources forthe imaging sensors, motion sensors, depth sensors, among othercomponents. Based on sensor data, the input device can capture handgestures and perform posture detection of an operator. In one aspect,the operator-enabled inputs may be derived from modifications ofexisting operator-enabled inputs provided with the VR/AR device 60.

In addition, a planned mapping in preparation for an ablation proceduremay be produced as well using the electroanatomic mapping devices 50 incombination with the VR/AR device 60, gathered from the collectedelectroanatomic data. In another aspect, virtual calipers (e.g., aruler) can be displayed by the VR/AR device 60 to allow the operator, inthe virtual environment, to make real time, accurate measurements. Thisfeature allows the operator to make measurements in various places-forinstance, when measuring electrograms, using milliseconds, and in thecardiac geometry, measuring in millimeters.

Other electroanatomic information can be displayed or communicated tothe operator. For example, the electroanatomic information can bedigitally represented on the 3D model 65 of the heart (e.g. color-codedareas on the heart to indicate electric activity and the force data ofthe EP catheter as it is applied), displayed visually to the operator(e.g., a table showing the relevant force data and electric activity),or in an auditory fashion. In an aspect, the auditory fashion can beused to inform the operator of the force that is being applied by the EPcatheter as it is activated during the ablation procedure. In suchinstances, the auditory response can be proportional to the force as itis applied. The auditory signal relates to the force being applied bythe ablation catheter. The stronger the force applied, the more frequentand high pitch the tone will sound for the operator. This auditoryfeature will only be present of force sensing catheters. An example of aforce sensing catheter is the TACTICATH′ (St Jude Medical), whichprovides feedback to the operator indicating how much force the tip ofthe catheter is applying to tissue (e.g. measured in grams).

FIGS. 5A, 5B, and 5C illustrate exemplary views of the holographicimages displayed by the VR/AR device 60 according to one aspect of thedisclosure. FIG. 5A illustrates a main menu with a 3D model 65, avirtual cardiac model, turned on. The transparency of the 3D cardiacmodel 65 in this view is increased to allow the operator to quicklyvisualize precise catheter location in one or more planes without theneed to change the model's orientation. In various aspects, thecomputing device 30 of the VR/AR system may segment the 3D model intosubunits to facilitate selection of portions of the 3D model 65 formodifying the holographic display by rendering a portion of the 3D model65 transparent and/or invisible. Non-limiting examples of subunits ofthe 3D model 65 that may be segmented by the computing device 30 includeleft and right atria, left and right ventricles, one or more valves, oneof more arteries and veins associated with the heart, and any otherrelevant cardiac structure.

FIG. 5B illustrates a 3D model 65 displayed in a posterior projectionwith the catheter locations turned on, with part of the heart cut awayusing a cutting plane feature so that intracardiac catheter locationscan be visualized. In this view, four catheters (circled) are seenpositioned in the coronary sinus (in the atrioventricular grooveseparating the left atrium from left ventricle; towards the left side ofthe screen), the high right atrium (in the upper rightward chamber, nearthe right atrial/superior vena cava junction), in the right ventricularapex (in the lower rightward chamber, pointing towards the apex of theheart) and the normal His-conduction system (towards the center, orcrux, of the heart). In some aspects, the computing device 30 includesadditional components in the displayed 3D model 65 based on anoperator-controlled level of zoom (e.g., magnification/reduction) of theholographic display. For example, at a greater level of zoom (i.e.,magnified image), the computing device 30 includes one or more of thefour catheters shown in FIG. 5B. On the other hand, at a lower level ofzoom (i.e., reduced image), the computing device 30 does not include thefour catheters, e.g., due to a resulting limited level of resolution ofthe displayed 3D model 65. To the right in FIG. 5B, the fluoroscopyscreen has been turned on in the virtual environment.

FIG. 5C shows the 3D model 65 displayed as oriented in a down-the-barrelview (i.e., a surgeon's view) of the ventricles, with the atria andgreat arteries virtually removed, e.g., a cross-sectional view. Theseimages provide non-limiting examples of potential views that an operatormay see using the VR/AR system 10, and should not be construed as theonly views provided by the system 10 of the disclosure.

In various aspects, the computing device 30 can include, but is notlimited to, laptop computers, desk top computers, tablets, servers witha connected display and the like. According to an aspect, as shown inFIG. 6, the computing device 30 can communicate with the 3D medicalimaging device 40, the electroanatomic mapping device 50, and the VR/ARdevice 60 through various known means, including wired and wirelessconnections known in the art. In an aspect, the computing device 30 caninclude a wireless interface controller (“W.I.”) 100 configured tocontrol the operation of the radio transceiver 102, as well as thereceiving and sending of information from the other devices. The radiotransceiver 102 may communicate on a wide range of public frequencies,including, but not limited to, frequency bands 2.40 Hz and/or 5 GHz-5.8GHz. In addition, the radio transceiver 102, with the assistance of thewireless interface controller 100, may also utilize a variety of publicprotocols. For example, in some embodiments of the disclosure, thecombination wireless interface controller 100 and radio transceiver 102may operate on various existing and proposed IEEE wireless protocols,including, but not limited to, IEEE 802.11 b/g/n/a/ac, with maximumtheoretical data transfer rates/throughput of 11 Mbps/54 Mbps/600Mbps/54 Mbps/1 Gbps respectively. In an aspect, the computing device 30may include a network adapter 126 configured to communicate with otherdevices over various networks and connections.

The computing device 30 may have one or more software applications 104to perform the methods discussed above. The computing device 30 includessystem memory 108, which can store the various applications 104,including applications to carry out functions discussed above, as wellas the operating system 110. The system memory 108 may also include data112 accessible by the various software applications 104. The systemmemory 108 can include random access memory (RAM) or read only memory(ROM). Data 112 stored on the computing device 30 may be any type ofretrievable data. The data may be stored in a wide variety of databases,including relational databases, including, but not limited to, MicrosoftAccess and SQL Server, MySQL, INGRES, DB2, INFORMIX, Oracle, PostgreSQL,Sybase 11, Linux data storage means, and the like.

The computing device 30 can include a variety of other computer readablemedia, including a storage device 114. The storage device 114 can beused for storing computer code, computer readable instructions, programmodules, and other data 112 for the computing device 30, and the storagedevice 114 can be used to back up or alternatively to run the operatingsystem 110 and/or other applications 104. The storage device 114 mayinclude a hard disk, various magnetic storage devices such as magneticcassettes or disks, solid-state flash drives, or other optical storage,random access memories, and the like.

The computing device 30 may include a system bus 118 that connectsvarious components of computing device 30 to the system memory 108 andto the storage device 114, as well as to each other. Other components ofthe computing device 30 may include one or more processors or processingunits 120, a user interface (U.I.) 122, and one or more input/outputinterfaces 124. In addition, the computing device 30 includes a networkadapter 126. In addition, the computing device 30 can included a powersource 128, including, but not limited to, a battery or an externalpower source. In addition, the computing device 30 can include a displayadapter 226 and a display 228 (e.g., a monitor or screen). In addition,input devices (e.g., key board, mouse, joy stick, etc.) can be used viathe input output interfaces 124. Further, the other 3D medical imagingdevice 40, electroanatomic mapping device 50, and VR/AR device 60 cancommunicate with the computing device 30 via the input/output interfaces124 as well.

The VR/AR system 10 is configured to display virtual, patient-specific3D models 65 in front of interventional physicians (e.g., operators)during diagnostic and/or surgical procedures. By using 3D models 65 toreveal real-time electrophysiology, improvements in physician training,patient outcomes, and clinician collaboration will occur, as well asdecrease radiation exposure rates to both the patient and physician.Further, the system 10 will also reduce the medical and economic burdenfor a patient who would otherwise undergo multiple procedures thatresult from poor visualization of their anatomy.

While the foregoing written description of the disclosure enables one ofordinary skill to make and use what is considered presently to be thebest mode thereof those of ordinary skill will understand and appreciatethe existence of variations, combinations, and equivalents of thespecific embodiment, method, and examples herein. The disclosure shouldtherefore not be limited by the above described embodiments, methods,and examples, but by all embodiments and methods within the scope andspirit of the disclosure. To the extent necessary to understand orcomplete the disclosure, all publications, patents, and patentapplications mentioned herein are expressly incorporated by referencetherein to the same extent as though each were individually soincorporated.

Having thus described exemplary embodiments of the disclosure, thoseskilled in the art will appreciate that the within disclosures areexemplary only and that various other alternatives, adaptations, andmodifications may be made within the scope of the disclosure.Accordingly, the disclosure is not limited to the specific embodimentsas illustrated herein.

What is claimed is:
 1. A VR/AR visualization system, comprising: a VR/ARdevice comprising a holographic display configured to display aholographic image to an operator; and a computing device operativelycoupled to the VR/AR device, the computing device comprising anon-volatile memory and a processor, wherein the computing device isconfigured to: receive at least one stored 3D image of a subject'sanatomy; receive at least one real-time 3D position of at least onesurgical instrument; register the at least one real-time 3D position ofthe at least one surgical instrument to correspond to the at least onestored 3D image of the subject's anatomy; and generate the holographicimage comprising the at least one real-time 3D position of the at leastone surgical instrument overlaid on the at least one 3D image of thesubject's anatomy.
 2. The system of claim 1, wherein the at least one 3Dposition of the at least one surgical instrument is received from atleast one instrument position sensor of a surgical instrument systemoperatively coupled to the computing device.
 3. The system of claim 1,wherein the at least one stored 3D image of the subject's anatomy isobtained using a medical imaging device selected from: a CT scanner, anMRI scanner, a PET scanner, an ultrasound imaging system, and anelectroanatomic mapping system.
 4. The system of claim 1, wherein theVR/AR device further comprises at least one sensor configured to detecta cue produced by the operator.
 5. The system of claim 1, wherein thecue is selected from one or more of a gesture, an eye movement, a voicecomment, a facial expression, and a head movement.
 6. The system ofclaim 1, wherein the at least one stored 3D image of the subject'sanatomy comprises a 3D image representing at least a portion of thesubject comprising an internal tissue or an internal organ.
 7. Thesystem of claim 1, wherein the computing device is further configured tomodify the holographic image in response to a cue produced by theoperator, the modification comprising at least one of a zoom, arotation, a translation, a generation of a cross section, an addition ofa portion of the at least one stored 3D image, a subtraction of at leasta portion of the at least one stored 3D image, and a change of renderingof the at least one stored 3D image.
 8. The system of claim 1, whereinthe computing device is further configured to update the holographicimage to incorporate changes in the received at least one real-time 3Dpositions of the at least one surgical instrument.
 9. The system ofclaim 1, wherein the computing device is further configured to receiveat least one additional data set and to generate the holographic imageto further comprise the at least one additional data set, wherein the atleast one additional data set is selected from at least one of: afluoroscopic image of the subject's anatomy, a patient vital signsdataset, and a patient demographic dataset.
 10. The system of claim 9,wherein the holographic image further comprises one or more additionaldisplay elements selected from a 2D fluoroscopic image and a data table,and wherein the one or more additional display elements are positionedanywhere within a field of view of the holographic display as specifiedby the operator.
 11. A method of VR/AR visualization of 3D medicalimages, the method comprising: receiving, using a computing device, atleast one stored 3D image of a subject's anatomy, the computing deviceoperatively coupled to a VR/AR device, the VR/AR device comprising aholographic display and at least one sensor; receiving, using thecomputing device, at least one real-time 3D position of at least onesurgical instrument; registering the at least one real-time 3D positionof the at least one surgical instrument to correspond to the at leastone stored 3D image of the subject's anatomy; and displaying, using theholographic display, a holographic image comprising the at least onereal-time 3D position of the at least one surgical instrument overlaidon the at least one 3D image of the subject's anatomy to an operator.12. The method of claim 11, wherein the at least one 3D position of theat least one surgical instrument is received from at least oneinstrument position sensor of a surgical instrument system operativelycoupled to the computing device.
 13. The method of claim 11, furthercomprising: detecting at least one cue produced by the operator via theat least one sensor of the VR/AR device; and modifying, using thecomputing device, the holographic image in response to the at least onecue.
 14. The method of claim 11, wherein the at least one stored 3Dimage of the subject's anatomy received by the computing device isobtained using a medical imaging device selected from: a CT scanner, anMM scanner, and a PET scanner, an ultrasound imaging system, and anelectroanatomic mapping system.
 15. The method of claim 11, furthercomprising manipulating the holographic image in response to detecting acue performed by the operator, wherein manipulating the holographicimage comprises at least one of zooming, rotating, a translating,generating a cross section, adding of a portion of the at least onestored 3D image, subtracting at least a portion of the at least onestored 3D image, and changing a rendering of the at least one stored 3Dimage.
 16. The method of claim 11, further comprising updating, usingthe computing device, the holographic image to incorporate changes inthe received at least one real-time 3D positions of the at least onesurgical instrument.
 17. The method of claim 11, further comprisingreceiving, using the computing device, at least one additional data setand generating, using the computing device, the holographic image tofurther comprise the at least one additional data set, wherein the atleast one additional data set is selected from at least one of: afluoroscopic image of the subject's anatomy, a patient vital signsdataset, and a patient demographic dataset.
 18. The method of claim 17,wherein the holographic image further comprises one or more additionaldisplay elements selected from a 2D fluoroscopic image and a data table,and wherein the one or more additional display elements are positionedanywhere within a field of view of the holographic display as specifiedby the operator.
 19. At least one non-transitory computer-readablestorage media for providing VR/AR visualization of three-dimensionalmedical images to an operator, the computer-readable storage mediahaving computer-executable instructions embodied thereon, wherein, whenexecuted by at least one processor, the computer-executable instructionscause the processor to: receive at least one stored 3D image of asubject's anatomy; receive at least one real-time 3D position of atleast one surgical instrument; register the at least one real-time 3Dposition of the at least one surgical instrument to correspond to the atleast one stored 3D image of the subject's anatomy; and display aholographic image comprising the at least one real-time 3D position ofthe at least one surgical instrument overlaid on the at least one 3Dimage of the subject's anatomy to an operator.
 20. The at least onenon-transitory computer-readable storage media of claim 19, wherein thecomputer-executable instructions further cause the processor to: detectat least one cue produced by the operator via the at least one sensor ofthe VR/AR device; and modify the holographic image in response to the atleast one cue.